Endoscope image processing device and endoscope image processing method

ABSTRACT

An endoscope image processing device includes a processor. The processor sequentially receives an observation image obtained by performing image pickup of an object inside a tubular portion of a subject, performs processing for detecting a region of interest with respect to the observation image, performs judgement processing as to whether degradation of visibility of the region of interest included in the observation image is predicted, and performs emphasis processing for emphasizing the position of the region of interest when a judgement result indicating that the degradation of visibility of the region of interest included in the observation image is predicted is acquired by the judgement processing.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of PCT/JP2017/019665filed on May 26, 2017, the entire contents of which are incorporatedherein by this reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an endoscope image processing deviceand an endoscope image processing method.

2. Description of the Related Art

In endoscope apparatuses, operators have conventionally determined thepresence or absence of lesioned parts, etc. by viewing observationimages. In order to prevent operators from overlooking lesioned partswhen observing observation images, an endoscope apparatus has beenproposed which displays an observation image while adding an alert imageto a region of interest detected by image processing as disclosed inJapanese Patent Application Laid-Open Publication No. 2011-255006.

SUMMARY OF THE INVENTION

An endoscope image processing device according to an aspect of thepresent invention includes a processor. The processor sequentiallyreceives an observation image obtained by performing image pickup of anobject inside a tubular portion of a subject, performs processing fordetecting a region of interest with respect to the observation image,performs judgement processing as to whether degradation of visibility ofthe region of interest included in the observation image is predicted,and performs emphasis processing for emphasizing the position of theregion of interest when a judgement result indicating that degradationof visibility of the region of interest included in the observationimage is predicted is obtained by the judgement processing.

An endoscope image processing method according to an aspect of thepresent invention performs processing for detecting a region of interestwith respect to an observation image obtained by performing image pickupof an object inside a tubular portion of a subject, performs judgementprocessing as to whether degradation of visibility of the region ofinterest included in the observation image is predicted, and performsemphasis processing for emphasizing the position of the region ofinterest when a judgement result indicating that degradation ofvisibility of the region of interest included in the observation imageis predicted is obtained by the judgement processing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing a configuration of a main part of anendoscope system including an endoscope image processing deviceaccording to an embodiment;

FIG. 2 is a block diagram showing an example of a specific configurationof the endoscope image processing device according to the embodiment;

FIG. 3 is a diagram showing an example of a display image which issubjected to processing of the endoscope image processing deviceaccording to the embodiment and then displayed on a display device;

FIG. 4 is a diagram showing an example of the processing performed inthe endoscope image processing device according to the embodiment;

FIG. 5A is a diagram showing an example of the display image which issubjected to the processing of the endoscope image processing deviceaccording to the embodiment and then displayed on the display device;

FIG. 5B is a diagram showing an example of the display image which issubjected to the processing of the endoscope image processing deviceaccording to the embodiment and then displayed on the display device;

FIG. 6A is a diagram showing an example of the display image which issubjected to the processing of the endoscope image processing deviceaccording to the embodiment and then displayed on the display device;

FIG. 6B is a diagram showing an example of the display image which issubjected to the processing of the endoscope image processing deviceaccording to the embodiment and then displayed on the display device;

FIG. 7A is a diagram showing an example of the display image which issubjected to the processing of the endoscope image processing deviceaccording to the embodiment and then displayed on the display device;and

FIG. 7B is a diagram showing an example of the display image which issubjected to the processing of the endoscope image processing deviceaccording to the embodiment and then displayed on the display device.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

An embodiment according to the present invention will be describedhereunder with reference to the drawings.

As shown in FIG. 1, an endoscope system 1 is configured to include anendoscope 21, a main body device 31, an endoscope image processingdevice 32, and a display device 41. FIG. 1 is a diagram showing aconfiguration of a main part of the endoscope system including theendoscope image processing device according to the embodiment.

The endoscope 21 is configured to be freely detachably connected to themain body device 31. The endoscope 21 is configured to include anelongated insertion portion 22 a that is insertable into a luminal organof a subject person, and an operation unit 22 b provided at a proximalend portion of the insertion portion 22 a. Furthermore, the endoscope 21is provided with a channel 22 c, one end of which is attachable to anddetachable from the main body device 31, and the other end of which isarranged at a distal end portion of the insertion portion 22 a.

A light source unit 23 and an image pickup unit 24 are provided at thedistal end portion of the insertion portion 22 a.

The light source unit 23 is configured to include a light emittingelement such as a white LED. The light source unit 23 is configured togenerate illumination light by emitting light according to a lightsource drive signal supplied from the main body device 31, and emit thegenerated illumination light to an object such as a living tissue.

The image pickup unit 24 is configured to include an image sensor suchas a color CCD or a color CMOS. The image pickup unit 24 is configuredto perform an operation responding to an image pickup control signaloutputted from the main body device 31. Furthermore, the image pickupunit 24 is configured to receive reflected light from the object whichis illuminated with the illumination light emitted from the light sourceunit 23, picks up an image of the received reflected light to generatean image pickup signal, and outputs the generated image pickup signal tothe main body device 31.

The operation unit 22 b is configured to have a shape that enables auser such as an operator to grip and operate the operation unit 22 b.Furthermore, the operation unit 22 b is provided with a scope switch 25having one or more switches for performing an instruction correspondingto a user's operation on the main body device 31.

For example, the scope switch 25 is provided with a water feeding switch(not shown) which is a push button switch capable of issuing aninstruction for causing water to be jetted to the front side of thedistal end portion of the insertion portion 22 a (hereinafter,abbreviated as a water feeding instruction), and an air feeding switch(not shown) which is a push button switch capable of issuing aninstruction for causing air to be jetted to the front side of the distalend portion of the insertion portion 22 a (hereinafter, abbreviated asan air feeding instruction).

The channel 22 c is configured as a conduit having an inner diameterwhich enables fluid (water and/or air) supplied from the main bodydevice 31 to flow through the channel 22 c. Furthermore, the channel 22c is configured by providing the distal end portion of the insertionportion 22 a with an opening portion (not shown) through which the fluidsupplied from the main body device 31 is jetted.

The main body device 31 is configured to be freely detachably connectedto each of the endoscope 21 and the endoscope image processing device32. The main body device 31 is configured to include a light sourcedriving device 31 a, a video processor 31 b, a water feeding device 31c, and an air feeding device 31 d.

The light source driving device 31 a is configured to include, forexample, a drive circuit. The light source driving device 31 a isconfigured to generate a light source drive signal for driving the lightsource unit 23 based on the light source control signal from the videoprocessor 31 b, and output the generated light source drive signal tothe endoscope 21.

The video processor 31 b is configured to generate a light sourcecontrol signal for controlling the light emission state of the lightsource unit 23 and output the light source control signal to the lightsource driving device 31 a. Furthermore, the video processor 31 b isconfigured to generate an image pickup control signal for controllingthe image pickup operation of the image pickup unit 24 and output thegenerated image pickup control signal to the endoscope 21. Stillfurthermore, the video processor 31 b is configured to performpredetermined processing on the image pickup signal outputted from theendoscope 21 to generate an observation image G1 of an object, andsuccessively output the generated observation image G1 to the endoscopeimage processing device 32 frame by frame.

The water feeding device 31 c is configured to include, for example, apump and a tank for water feeding. Furthermore, the water feeding device31 c is configured to perform an operation for supplying water stored inthe tank to the channel 22 c when detecting a water feeding instructionfrom the operation unit 22 b. In addition, the water feeding device 31 cis configured to generate a water feeding instruction detection signalwhich is a signal indicating the presence or absence of a water feedinginstruction from the operation unit 22 b and output the generated waterfeeding instruction detection signal to the endoscope image processingdevice 32.

The air feeding device 31 d is configured to include, for example, apump and a bomb for air feeding. Furthermore, the air feeding device 31d is configured to perform an operation for feeding air stored in thebomb to the channel 22 c when detecting an air feeding instruction fromthe operation unit 22 b. In addition, the air feeding device 31 d isconfigured to generate an air feeding instruction detection signal whichis a signal indicating the presence or absence of an air feedinginstruction from the operation unit 22 b, and output the generated airfeeding instruction detection signal to the endoscope image processingdevice 32.

The endoscope image processing device 32 is configured to be freelydetachably connected to each of the main body device 31 and the displaydevice 41. Furthermore, the endoscope image processing device 32 isconfigured to generate a display image based on the observation image G1outputted from the main body device 31 (video processor 31 b), andperform an operation of causing the display device 41 to display thegenerated display image. In addition, as shown in FIG. 2, the endoscopeimage processing device 32 is configured to include a region-of-interestdetection unit 34, a visibility judgement unit 35, and a display controlunit 36. Note that according to the present embodiment, for example,each unit of the endoscope image processing device 32 may be configuredas an individual electronic circuit, or configured as a circuit block inan integrated circuit such as FPGA (field programmable gate array).According to the present embodiment, for example, at least a part of theendoscope image processing device 32 may be configured by CPU. FIG. 2 isa block diagram showing an example of a specific configuration of theendoscope image processing device according to the embodiment.

The region-of-interest detection unit 34 is configured to calculate apredetermined feature value related to the observation image G1sequentially outputted from the main body device 31, and further detect,based on the calculated predetermined feature value, a lesion candidateregion L that is a region of interest included in the observation imageG1. In other words, the region-of-interest detection unit 34 isconfigured to sequentially receive a plurality of observation images G1obtained by picking up images of an object with the endoscope 21, andperform processing for detecting a lesion candidate region L for each ofthe plurality of observation images G1. Furthermore, as shown in FIG. 2,the region-of-interest detection unit 34 is configured to include afeature value calculation unit 34 a and a lesion candidate detectionunit 34 b.

The feature value calculation unit 34 a is configured to calculate apredetermined feature value related to the observation image G1sequentially outputted from the main body device 31, and output thecalculated predetermined feature value to the lesion candidate detectionunit 34 b.

Specifically, for example, the observation image G1 is divided into aplurality of small regions each having a predetermined size, and thefeature value calculation unit 34 a calculates, as a feature value foreach small region of the plurality of small regions, an inclinationvalue which is a value indicating the amount of variation in brightnessor the amount of variation in density in each pixel within one smallregion out of the plurality of small regions and each pixel within asmall region adjacent to the one small region. Note that the featurevalue calculation unit 34 a may calculate a value different from theabove-described inclination value as the feature value as long as thefeature value calculation unit 34 a calculates a quantitativelyevaluable value for the observation image G1.

The lesion candidate detection unit 34 b is configured to include anon-volatile memory (not shown) such as a ROM in which one or morepieces of polyp model information are stored in advance.

Specifically, the polyp model information stored in the memory of thelesion candidate detection unit 34 b is configured to include, forexample, feature values obtained by quantifying common points and/orsimilar points in a large number of polyp images.

The lesion candidate detection unit 34 b is configured to detect alesion candidate region L based on a predetermined feature valueoutputted from the feature value calculation unit 34 a and a pluralityof pieces of polyp model information read out from the memory, acquirelesion candidate information IL which is information indicating thedetected lesion candidate region L, and output the acquired lesioncandidate information IL to each of the visibility judgement unit 35 andthe display control unit 36.

Specifically, for example, when the feature value of one small regionoutputted from the feature value calculation unit 34 a is coincidentwith at least one feature value included in the plurality of pieces ofpolyp model information read out from the memory, the lesion candidatedetection unit 34 b detects the one small region as the lesion candidateregion L. Furthermore, the lesion candidate detection unit 34 b acquiresthe lesion candidate information IL including position information andsize information of the lesion candidate region L detected by theabove-described method, and outputs the acquired lesion candidateinformation IL to each of the visibility judgement unit 35 and thedisplay control unit 36.

Note that the position information of the lesion candidate region L isinformation indicating the position of the lesion candidate region L inthe observation image G1, and is acquired, for example, as pixelposition of the lesion candidate region L existing in the observationimage G1. Furthermore, the size information of the lesion candidateregion L is information indicating the size of the lesion candidateregion L in the observation image G1, and is acquired, for example, asthe number of pixels of the lesion candidate region L existing in theobservation image G1.

As long as the region-of-interest detection unit 34 performs theprocessing for detecting the lesion candidate region L from theobservation image G1, the region-of-interest detection unit 34 may notbe configured to include the feature value calculation unit 34 a and thelesion candidate detection unit 34 b. Specifically, theregion-of-interest detection unit 34 may be configured to detect thelesion candidate region L from the observation image G1, for example, byperforming processing of applying, to the observation image G1, an imagediscriminator which has beforehand acquired a function capable ofidentifying a polyp image by a learning method such as deep learning.

Based on the observation image G1 sequentially outputted from the mainbody device 31, the water feeding instruction detection signal and theair feeding instruction detection signal outputted from the main bodydevice 31, and the lesion candidate information IL outputted from thelesion candidate detection unit 34 b, the visibility judgement unit 35is configured to perform judgement processing as to whether degradationof visibility of the lesion candidate region L corresponding to thelesion candidate information IL is predicted, and output a judgementresult obtained by the judgement processing to the display control unit36. Note that specific processing to be performed in the visibilityjudgement unit 35 will be described later.

The display control unit 36 is configured to perform processing forgenerating a display image by using the observation image G1sequentially outputted from the main body device 31, and performprocessing of causing the generated display image to be displayed on thedisplay screen of the display device 41. The display control unit 36 isconfigured to include an emphasis processing unit 36 a for performingemphasis processing to emphasize the lesion candidate region L existingin the observation image G1.

The emphasis processing unit 36 a is configured to perform, as emphasisprocessing, processing for generating a marker image G2 for emphasizingthe position of the lesion candidate region L existing in theobservation image G1 based on the lesion candidate information IL andadding the marker image G2 to the observation image G1 when apredetermined judgement result (described later) is outputted from thevisibility judgement unit 35.

Note that the marker image G2 added by the emphasis processing of theemphasis processing unit 36 a may have any form as long as the markerimage G2 can present the position of the lesion candidate region L asvisual information. In other words, as long as the emphasis processingunit 36 a generates the marker image G2 for emphasizing the position ofthe lesion candidate region L, the emphasis processing unit 36 a mayperform the emphasis processing by using only the position informationincluded in the lesion candidate information IL. Alternatively, theemphasis processing unit 36 a may perform the emphasis processing byusing both the position information and the size information included inthe lesion candidate information IL.

The display device 41 is configured to include a monitor and the like,and configured to be capable of displaying the display image outputtedfrom the endoscope image processing device 32 on the display screen.

Subsequently, an operation of the present embodiment will be described.For the sake of simplification, the following description will be madeby exemplifying a case where one lesion candidate region L1 is includedin the observation image G1. The following operation is suitably appliedin such a case that the lesion candidate region L1 exists on an innercircumferential surface of a luminal organ, that is, a region ofinterest exists on an inner circumferential surface of a tubular portionof a subject.

The user connects each part of the endoscope system 1 and turns on thepower, and then performs an operation for inserting the insertionportion 22 a into a luminal organ of the subject person. In other words,the insertion portion 22 a is inserted into the inside of the tubularportion of the subject by a user's manipulation.

With turn-on of a power source of the main body device 31, the endoscope21 emits illumination light to an object, receives reflected light fromthe object, picks up an image of the received reflected light togenerate an image pickup signal, and outputs the generated image pickupsignal to the main body device 31. In other words, the endoscope 21outputs, to the main body device 31, an image pickup signal obtained bypicking up an image of the object inside a tubular portion of thesubject.

The main body device 31 (video processor 31 b) generates an observationimage G1 of the object by performing predetermined processing on theimage pickup signal outputted from the endoscope 21, and sequentiallyoutputs the generated observation image G1 frame by frame to theendoscope image processing device 32.

The lesion candidate detection unit 34 b detects a lesion candidateregion L1 included in the observation image G1 to acquire lesioncandidate information IL1 that is information indicating the detectedlesion candidate region L1, and outputs the acquired lesion candidateinformation IL1 to each of the visibility judgement unit 35 and thedisplay control unit 36.

When the lesion candidate information IL1 is not outputted from thelesion candidate detection unit 34 b, the display control unit 36performs processing for causing the observation image G1 outputted fromthe main body device 31 to be displayed as a display image.

When the lesion candidate information IL1 has been outputted from thelesion candidate detection unit 34 b and a judgement result indicatingthat degradation of visibility of the lesion candidate region L1included in the observation image G1 is not predicted is outputted fromthe visibility judgement unit 35, for example, as shown in FIG. 3, thedisplay control unit 36 performs processing of causing the observationimage G1 outputted from the main body device 31 to be displayed as adisplay image without causing the emphasis processing unit 36 a toperform the emphasis processing. FIG. 3 is a diagram showing an exampleof the display image which is subjected to the processing of theendoscope image processing device according to the present embodimentand then displayed on the display device.

When the lesion candidate information IL1 has been outputted from thelesion candidate detection unit 34 b and a judgement result indicatingthat degradation of visibility of the lesion candidate region L1included in the observation image G1 is predicted is outputted from thevisibility judgement unit 35, the display control unit 36 performsprocessing of causing the observation image G1 added with a marker imageG2 by the emphasis processing of the emphasis processing unit 36 a to bedisplayed as a display image. In other words, when the judgement resultindicating that the degradation of visibility of the lesion candidateregion L1 included in the observation image G1 is predicted is outputtedfrom the visibility judgement unit 35, the emphasis processing unit 36 aperforms emphasis processing of emphasizing the position of the lesioncandidate region L1 existing in the observation image G1 based on thelesion candidate information IL.

When the lesion candidate information IL1 is outputted from the lesioncandidate detection unit 34 b, the visibility judgement unit 35 performsdetermination processing as to whether degradation of visibility of thelesion candidate region L1 corresponding to the lesion candidateinformation IL1 is predicted.

Here, a specific example of the judgement processing performed by thevisibility judgement unit 35 of the present embodiment will be describedwith reference to FIG. 4 and the like. FIG. 4 is a diagram showing anexample of processing performed in the endoscope image processing deviceaccording to the embodiment.

The visibility judgement unit 35 performs processing of detecting thepresence or absence of water feeding to the lesion candidate region L1included in the observation image G1 based on the observation image G1sequentially outputted from the main body device 31, the water feedinginstruction detection signal outputted from the main body device 31, andthe lesion candidate information IL1 outputted from the lesion candidatedetection unit 34 b (step S1 in FIG. 4).

When the visibility judgement unit 35 detects water feeding to thelesion candidate region L1 included in the observation image G1 (S1:YES), the visibility judgement unit 35 acquires a judgement resultindicating that the degradation of visibility of the lesion candidateregion L1 is predicted, and outputs the judgement result to the displaycontrol unit 36 (step S10 of FIG. 4). When the visibility judgement unit35 cannot detect water feeding to the lesion candidate region L1included in the observation image G1 (S1: NO), the visibility judgementunit 35 proceeds to perform processing of step S2 in FIG. 4 describedlater.

Specifically, for example, when the water feeding instruction detectionsignal is outputted from the main body device 31, based on a processingresult obtained by performing predetermined image processing on theobservation image G1 and the lesion candidate information IL1 outputtedfrom the lesion candidate detection unit 34 b, the visibility judgementunit 35 detects whether water jetted from the distal end portion of theinsertion portion 22 a is applied to the lesion candidate region L1 or aneighborhood of the lesion candidate region L1. Then, when thevisibility judgement unit 35 detects that the water jetted from thedistal end portion of the insertion portion 22 a is applied to thelesion candidate region L1 included in the observation image G1 or theneighborhood of the lesion candidate region L1, the visibility judgementunit 35 acquires the judgement result indicating that the degradation ofvisibility of the lesion candidate region L1 is predicted. Furthermore,when the visibility judgement unit 35 detects that the water jetted fromthe distal end portion of the insertion portion 22 a is applied toneither the lesion candidate region L1 included in the observation imageG1 nor the neighborhood of the lesion candidate region L1, thevisibility judgement unit 35 proceeds to perform the processing of stepS2 of FIG. 4 described later.

Note that the visibility judgement unit 35 may acquire a judgment resultindicating that degradation of visibility of the lesion candidate regionL1 included in the observation image G1 is predicted, for example, whenthe output of the water feeding instruction detection signal from themain body device 31 is detected.

When the judgment result indicating that the degradation of visibilityof the lesion candidate region L1 is predicted is obtained by theprocessing of step S1 in FIG. 4, the emphasis processing for adding theobservation image G1 with the marker image G2 is started, for example,at a timing either when water jetted from the distal end portion of theinsertion portion 22 a is applied to the neighborhood of the lesioncandidate region L1 included in the observation image G1 (see FIG. 5A)or just after the water feeding switch of the operation unit 22 b ispushed. Furthermore, when the judgement result indicating that thedegradation of visibility of the lesion candidate region L1 is predictedis obtained by the processing of step S1 of FIG. 4, the emphasisprocessing for adding the observation image G1 with the marker image G2is continued, for example, either during a time period when water jettedfrom the distal end portion of the insertion portion 22 a is applied toat least a part of the lesion candidate region L1 included in theobservation image G1 (FIG. 5B) or a time period when the water feedingswitch of the operation unit 22 b is pushed. FIG. 5A and FIG. 5B arediagrams showing an example of a display image which is subjected to theprocessing of the endoscope image processing device according to thepresent embodiment and then displayed on the display device.

The visibility judgement unit 35 performs processing for detecting thepresence or absence of air feeding to the lesion candidate region L1included in the observation image G1 based on the air feedinginstruction detection signal outputted from the main body device 31(step S2 of FIG. 4).

When the visibility judgement unit 35 detects air feeding to the lesioncandidate region L1 included in the observation image G1 (S2: YES), thevisibility judgement unit 35 acquires a judgement result indicating thatdegradation of visibility of the lesion candidate region L1 ispredicted, and outputs the judgement result to the display control unit36 (step S10 in FIG. 4). When the visibility judgement unit 35 cannotdetect air feeding to the lesion candidate region L1 included in theobservation image G1 (S2: NO), the visibility judgement unit 35 proceedsto perform the processing of step S3 in FIG. 4 described later.

When the judgement result indicating that the degradation of visibilityof the lesion candidate region L1 is predicted is obtained by theprocessing of step S2 in FIG. 4, the emphasis processing for adding theobservation image G1 with the marker image G2 is started, for example,at a timing just after the air feeding switch of the operation unit 22 bis pushed. Furthermore, when the judgement result indicating that thedegradation of visibility of the lesion candidate region L1 is predictedis obtained by the processing of step S2 of FIG. 4, the emphasisprocessing for adding the observation image G1 with the marker image G2is continued, for example, during a time period when the air feedingswitch of the operation unit 22 b continues to be pushed.

Based on the processing result obtained by performing the predeterminedimage processing on the observation image G1 sequentially outputted fromthe main body device 31 and the lesion candidate information IL1outputted from the lesion candidate detection unit 34 b, the visibilityjudgement unit 35 performs processing of detecting the presence orabsence of an obstacle which is an object covering the entire lesioncandidate region L1 in the observation image G1, such as a fold, aresidue, blood, a treatment instrument or the like in a large intestine(step S3 of FIG. 4).

When the visibility judgement unit 35 detects the presence of anobstacle in the observation image G1 (S3: YES), the visibility judgementunit 35 proceeds to perform the processing of step S4 in FIG. 4described later. When the visibility judgement unit 35 cannot detect thepresence of an obstacle in the observation image G1 (S3: NO), thevisibility judgement unit 35 proceeds to perform the processing of stepS5 in FIG. 4 described later.

Based on a motion vector (change in position) of the obstacle detectedby the processing of step S3 in FIG. 4 and a motion vector (change inposition) of the center of gravity of the lesion candidate region L1obtained based on the position information included in the lesioncandidate information IL1 outputted from the lesion candidate detectionunit 34 b, the visibility judgement unit 35 performs processing fordetecting whether the lesion candidate region L1 is approaching theobstacle (step S4 of FIG. 4).

When the visibility judgement unit 35 detects that the lesion candidateregion L1 is approaching the obstacle detected by the processing of stepS3 in FIG. 4 (S4: YES), the visibility judgement unit 35 acquires ajudgement result indicating that degradation of visibility of the lesioncandidate region L1 is predicted, and outputs the judgement result tothe display control unit 36 (step S10 in FIG. 4). Furthermore, when thevisibility judgement unit 35 detects either that the lesion candidateregion L1 is being away from the obstacle detected by the processing ofstep S3 in FIG. 4 or that the position of the lesion candidate region L1has not changed (S4: NO), the visibility judgement unit 35 proceeds toperform processing of step S5 in FIG. 4 described later. In other words,when the visibility judgement unit 35 detects such a positional changeof the lesion candidate region L1 that the lesion candidate region L1 isapproaching the obstacle which covers the entire lesion candidate regionL1 in the observation image G1, the visibility judgement unit 35acquires a judgement result indicating that degradation of visibility ofthe lesion candidate region L1 included in the observation image G1 ispredicted.

When the judgement result indicating that the degradation of visibilityof the lesion candidate region L1 is predicted is acquired by theprocessing of step S4 in FIG. 4, the emphasis processing of adding theobservation image G1 with the marker image G2 is started, for example,at a timing when the lesion candidate region L1 included in theobservation image G1 moves to a position adjacent to the obstacle in theobservation image G1 (see FIG. 6A). When the judgement result indicatingthat the degradation of visibility of the lesion candidate region L1 ispredicted is acquired by the processing of step S4 in FIG. 4, theemphasis processing for adding the observation image G1 with the markerimage G2 is continued, for example, during a time period when at least apart of the lesion candidate region L1 is covered by the obstacle in theobservation image G1 (see FIG. 6B). FIGS. 6A and 6B are diagrams showingan example of a display image which is subjected to the processing ofthe endoscope image processing device according to the embodiment andthen displayed on the display device.

Note that in the present embodiment, when the emphasis processing foradding the marker image G2 is started after the processing of step S4 ofFIG. 4, the emphasis processing may be stopped, for example, at a timingwhen the entire lesion candidate region L1 included in the observationimage G1 is covered by the obstacle in the observation image G1, or theemphasis processing may be stopped after a predetermined time haselapsed since the entire lesion candidate region L1 is covered by theobstacle.

Based on the observation image G1 sequentially outputted from the mainbody device 31 and the motion vector (change in position) of the centerof gravity of the lesion candidate region L1 obtained based on theposition information included in the lesion candidate information IL1outputted from the lesion candidate detection unit 34 b, the visibilityjudgement unit 35 performs processing of detecting whether the lesioncandidate region L1 is approaching an end of the observation image G1(step S5 in FIG. 4).

When the visibility judgement unit 35 detects that the lesion candidateregion L1 is approaching the end of the observation image G1 (S5: YES),the visibility judgement unit 35 acquires a judgement result indicatingthat degradation of visibility of the lesion candidate region L1 ispredicted, and outputs the judgement result to the display control unit36 (step S10 in FIG. 4). When the visibility judgement unit 35 detectseither that the lesion candidate region L1 is being away from the end ofthe observation image G1 or that the position of the lesion candidateregion L1 has not changed (S5: NO), the visibility judgement unit 35proceeds to perform processing of step S6 of FIG. 4 described later. Inother words, when the visibility judgement unit 35 detects such apositional change of the lesion candidate region L1 that the lesioncandidate region L1 is approaching the end of the observation image G1,the visibility judgement unit 35 acquires a judgement result indicatingthat the degradation of visibility of the lesion candidate region L1included in the observation image G1 is predicted.

When the judgement result indicating that the degradation of visibilityof the lesion candidate region L1 is predicted is acquired by theprocessing of step S5 of FIG. 4, the emphasis processing for adding theobservation image G1 with the marker image G2 is started, for example,at a timing when the lesion candidate region L1 included in theobservation image G1 enters a predetermined region corresponding to anend portion of the observation image G1 (see FIG. 7A). Furthermore, whenthe judgement result indicating that the degradation of visibility ofthe lesion candidate region L1 is predicted is acquired by theprocessing of step S5 in FIG. 4, the emphasis processing for adding theobservation image G1 with the marker image G2 is continued, for example,during a time period when at least a part of the lesion candidate regionL1 exists in the predetermined region corresponding to the end portionof the observation image G1 (see FIG. 7B). FIGS. 7A and 7B are diagramsshowing an example of a display image which is subjected to theprocessing of the endoscope image processing device according to theembodiment, and then displayed on the display device.

The visibility judgement unit 35 performs processing for detectingwhether the insertion portion 22 a is moving in a direction in which theinsertion portion 22 a is removed from the inside of a luminal organbased on the observation image G1 sequentially outputted from the mainbody device 31 (step S6 in FIG. 4).

Specifically, the visibility judgement unit 35 performs processing usingan optical flow of the observation image G1 which is a vector fieldconfigured by motion vectors of all pixels included in the observationimage G1 sequentially outputted from the main body device 31, therebydetecting whether the insertion portion 22 a is moving in a direction inwhich the insertion portion 22 a is removed from the inside of theluminal organ.

When the visibility judgement unit 35 detects that the insertion portion22 a is moving in the direction in which the insertion portion 22 a isremoved from the inside of the luminal organ (S6: YES), the visibilityjudgement unit 35 acquires the judgement result indicating thatdegradation of visibility of the lesion candidate region L1 ispredicted, and outputs the judgement result to the display control unit36 (step S10 in FIG. 4). When the visibility judgement unit 35 detectseither that the insertion portion 22 a is moving in a direction in whichthe insertion portion 22 a is inserted into a deep part of the luminalorgan or that the insertion portion 22 a is stopped (S6: NO), thevisibility judgement unit 35 proceeds to perform processing of step S7in FIG. 4 described later.

When the judgement result indicating that the degradation of visibilityof the lesion candidate region L1 is predicted is acquired by theprocessing of step S6 in FIG. 4, the emphasis processing for adding theobservation image G with the marker image G2 is started, for example, ata timing when the lesion candidate region L1 included in the observationimage G1 starts to move from a front side to a back side in theobservation image G1. Furthermore, when the judgement result indicatingthat the degradation of visibility of the lesion candidate region L1 ispredicted is acquired by the processing of step S6 of FIG. 4, theemphasis processing for adding the observation image G1 with the markerimage G2 is continued, for example, during a time period when the lesioncandidate region L1 continues to move from the front side to the backside in the observation image G1.

Note that according to the present embodiment, the processing asdescribed above is not limited to the processing performed in step S6 ofFIG. 4, and, for example, such processing as detects, based on theoptical flow of the observation image G1 and the lesion candidateinformation IL1 outputted from the lesion candidate detection unit 34 b,whether the lesion candidate region L1 included in the observation imageG1 is moving from the front side to the back side in the observationimage G1 may be performed in step S6 in FIG. 4. It may be set for suchprocessing that when it is detected that the lesion candidate region L1is moving from the front side to the back side in the observation imageG1, a judgement result indicating that degradation of visibility of thelesion candidate region L1 is predicted is acquired.

Based on the observation image G1 sequentially outputted from the mainbody device 31 and the lesion candidate information IL1 outputted fromthe lesion candidate detection unit 34 b, the visibility judgement unit35 performs processing for detecting whether the brightness of thelesion candidate region L1 included in the observation image G1 hasdecreased and reached a predetermined brightness (step S7 in FIG. 4).

When the visibility judgement unit 35 detects that the brightness of thelesion candidate region L1 has decreased and reached the predeterminedbrightness (S7: YES), the visibility judgement unit 35 acquires ajudgement result indicating that the degradation of visibility of thelesion candidate region L1 is predicted, and outputs the judgementresult to the display control unit 36 (step S10 in FIG. 4). When thevisibility judgement unit 35 detects either that the brightness of thelesion candidate region L1 has increased or that the brightness of thelesion candidate region L1 has not decreased until it has reached thepredetermined brightness (step S5: NO), the visibility judgement unit 35proceeds to perform processing of step S8 in FIG. 4 described later.

Specifically, the visibility judgement unit 35 performs, for example,processing of specifying the lesion candidate region L1 in theobservation image G1 based on the position information included in thelesion candidate information ILL and also detecting whether an averagebrightness value AVB of respective pixels of the specified lesioncandidate region L1 has decreased and reached a predetermined brightnessvalue TB. When the visibility judgement unit 35 detects that the averagebrightness value AVB of the respective pixels of the lesion candidateregion L1 has decreased and reached the predetermined brightness valueTB, the visibility judgement unit 35 acquires a judgement resultindicating that the degradation of visibility of the lesion candidateregion L1 is predicted. Furthermore, when the visibility judgement unit35 detects that the average brightness value AVB has increased or theaverage brightness value AVB has not decreased until the averagebrightness value AVB has reached the predetermined brightness value TB,the visibility judgement unit 35 proceeds to perform the processing ofstep S8 of FIG. 4 described later. Note that the predeterminedbrightness value TB may be appropriately set, for example, as abrightness value corresponding to a lower limit of brightness whichmakes it possible to secure the visibility of the lesion candidateregion L1 included in the observation image G1.

Note that according to the present embodiment, for example, processingfor detecting whether the brightness of a neighborhood of the lesioncandidate region L1 in the observation image G1 has decreased andreached a predetermined brightness may be performed in step S7 of FIG.4. Furthermore, when it is detected according to the processing asdescribed above that the brightness of the neighborhood of the lesioncandidate region L1 in the observation image G1 has decreased andreached the predetermined brightness, a judgement result indicating thatthe degradation of visibility of the lesion candidate region L1 ispredicted may be acquired. According to the present embodiment, forexample, processing for detecting whether a contrast value of the lesioncandidate region L1 included in the observation image G1 has decreasedand reached a predetermined contrast value may be performed in step S7of FIG. 4. Furthermore, when it is detected according to the processingas described above that the contrast value of the lesion candidateregion L1 included in the observation image G1 has decreased and reachedthe predetermined contrast value, a judgement result indicating that thedegradation of visibility of the lesion candidate region L1 is predictedmay be acquired. According to the present embodiment, for example,processing for detecting whether the contrast value of a neighborhood ofthe lesion candidate region L1 in the observation image G1 has decreasedand reached a predetermined contrast value may be performed in step S7of FIG. 4. When it is detected according to the processing as describedabove that the contrast value of the neighborhood of the lesioncandidate region L1 in the observation image G1 has decreased andreached the predetermined contrast value, a judgement result indicatingthat the degradation of visibility of the lesion candidate region L1 ispredicted may be acquired.

The visibility judgement unit 35 performs processing for detectingwhether the size has reached a predetermined size with decreasing of thesize of the lesion candidate region L1 included in the observation imageG1 (the shape changes) based on the size information included in thelesion candidate information IL1 outputted from the lesion candidatedetection unit 34 b (step S8 in FIG. 4).

When the visibility judgement unit 35 detects that the size of thelesion candidate region L1 has decreased and reached a predeterminedsize (S8: YES), the visibility judgement unit 35 acquires a judgementresult indicating that degradation of visibility of the lesion candidateregion L1 is predicted, and outputs the judgement result to the displaycontrol unit 36 (step S10 in FIG. 4). When the visibility judgement unit35 detects that the size of the lesion candidate region L1 has increasedor the size of the lesion candidate region L1 has not decreased untilthe size of the lesion candidate region L1 has reached the predeterminedsize (S8: NO), the visibility judgement unit 35 acquires a judgementresult indicating that the degradation of visibility of the lesioncandidate region L1 is not predicted, and outputs the judgement resultto the display control unit 36 (step S9 in FIG. 4).

Note that according to the present embodiment, for example, processingfor detecting whether the shape of the lesion candidate region L1included in the observation image G1 has changed to a predeterminedshape may be performed in step S8 of FIG. 4. Furthermore, when it isdetected according to the processing as described above that the shapeof the lesion candidate region L1 included in the observation image G1has changed to a predetermined shape, a judgment result indicating thatthe degradation of visibility of the lesion candidate region L1 ispredicted may be acquired.

In the present embodiment, as long as it is possible to acquire ajudgement result as to whether degradation of visibility of the lesioncandidate region L1 is predicted, for example, some of the respectiveprocessing exemplified in FIG. 4 may be omitted as appropriate, theorder of the respective processing exemplified in FIG. 4 may berearranged as appropriate, or two or more of the respective processingexemplified in FIG. 4 may be performed in parallel.

In other words, based on at least one of the change of the lesioncandidate region L1 included in the observation image G1 and the changein a region in the neighborhood of the lesion candidate region L1, thevisibility judgement unit 35 of the present embodiment performsjudgement processing as to whether degradation of visibility of thelesion candidate region L1 is predicted. Specifically, the visibilityjudgement unit 35 of the present embodiment performs the judgementprocessing as to whether the degradation of visibility of the lesioncandidate region L1 is predicted, based on a detection result obtainedby detecting at least any of the change in position of the lesioncandidate region L1 included in the observation image G1, the change inbrightness of the lesion candidate region L1, the change in shape of thelesion candidate region L1, the change in contrast of the lesioncandidate region L1, and the change in size of the lesion candidateregion L1.

Based on at least one of a function state and an operation state of theendoscope 21 configured to pick up an image of an object including alesion candidate region L1, the visibility judgement unit 35 of thepresent embodiment performs judgement processing as to whetherdegradation of visibility of the lesion candidate region L1 ispredicted. Specifically, the visibility judgement unit 35 of the presentembodiment performs the judgement processing as to whether degradationof visibility of a lesion candidate region L1 is predicted, based on adetection result obtained by detecting at least any one of the presenceor absence of water feeding from the endoscope 21 to the lesioncandidate region L1, the presence or absence of air feeding from theendoscope 21 to the lesion candidate region L1, and a moving directionof the insertion portion 22 a to be inserted into a luminal organ in theendoscope 21.

As described above, according to the present embodiment, whendegradation of visibility of a lesion candidate region L included in anobservation image G1 is predicted, it is possible to display a displayimage in which a marker image G2 for emphasizing the position of thelesion candidate region L is added to the observation image G1.Therefore, according to the present embodiment, even in a situationwhere visibility of a region of interest such as a lesioned part or thelike existing in an observation image may degrade, it is possible toprevent oversight of the region of interest. Furthermore, according tothe present embodiment, for example, when a lesion candidate region Lexists on an inner peripheral surface of a luminal organ, that is, whenthe lesion candidate region L included in an observation image G1 and aregion around the lesion candidate region in the observation image G1move simultaneously together while following the insertion and removalof the insertion portion 22 a, it is possible to prevent the lesioncandidate region L from being overlooked.

As described above, according to the present embodiment, whendegradation of visibility of a lesion candidate region L included in anobservation image G1 is not predicted, a marker image G2 for emphasizingthe position of the lesion candidate region L is not added to theobservation image G1. Therefore, according to the present embodiment,for example, when a user performs a work for visually searching for thepresence or absence of a lesioned part in an observation image, it ispossible not to hinder achievement of the skill meeting the work, andalso not to lower user's motivation for the work.

Note that the present invention is not limited to the above-describedembodiment, and it goes without saying that various modifications andapplications can be made without departing from the subject matter ofthe invention.

The endoscope image processing device and the like according to thepresent embodiment may include a processor and a storage (e.g., amemory). The functions of individual units in the processor may beimplemented by respective pieces of hardware or may be implemented by anintegrated piece of hardware, for example. The processor may includehardware, and the hardware may include at least one of a circuit forprocessing digital signals and a circuit for processing analog signals,for example. The processor may include one or a plurality of circuitdevices (e.g., an IC) or one or a plurality of circuit elements (e.g., aresistor, a capacitor) on a circuit board, for example. The processormay be a CPU (central processing unit), for example, but this should notbe construed in a limiting sense, and various types of processorsincluding a GPU (graphics processing unit) and a DSP (digital signalprocessor) may be used. The processor may be a hardware circuit with anASIC (application specific integrated circuit) or an FPGA(field-programmable gate array). The processor may include anamplification circuit, a filter circuit, or the like for processinganalog signals. The memory may be a semiconductor memory such as an SRAMand a DRAM; a register; a magnetic storage device such as a hard diskdevice; and an optical storage device such as an optical disk device.The memory stores computer-readable instructions, for example. When theinstructions are executed by the processor, the functions of each unitof the image processing device and the like are implemented. Theinstructions may be a set of instructions constituting a program or aninstruction for causing an operation on the hardware circuit of theprocessor.

The units in the endoscope image processing device and the like and thedisplay apparatus according to the present embodiment may be connectedwith each other via any types of digital data communication such as acommunication network or via communication media. The communicationnetwork may include a LAN (local area network), a WAN (wide areanetwork), and computers and networks which form the internet, forexample.

What is claimed is:
 1. An endoscope image processing device comprising aprocessor, the processor sequentially receiving an observation imageobtained by performing image pickup of an object inside a tubularportion of a subject, and performing processing for detecting a regionof interest with respect to the observation image; performing judgementprocessing as to whether degradation of visibility of the region ofinterest included in the observation image is predicted; and performingemphasis processing for emphasizing a position of the region of interestwhen a judgement result indicating that the degradation of visibility ofthe region of interest included in the observation image is predicted isacquired by the judgement processing.
 2. The endoscope image processingdevice according to claim 1, wherein the processor performs thejudgement processing based on at least one of change of the region ofinterest included in the observation image and change of a region nearthe region of interest.
 3. The endoscope image processing deviceaccording to claim 1, wherein the processor performs the judgementprocessing based on at least one of a function state and an operationstate of an endoscope configured to pick up an image of the objectincluding the region of interest.
 4. The endoscope image processingdevice according to claim 2, wherein the processor performs thejudgement processing based on a detection result obtained by detectingat least any one of change in position of the region of interest, changein brightness of the region of interest, change in brightness of aneighborhood of the region of interest, change in shape of the region ofinterest, change in contrast of the region of interest, change incontrast of a neighborhood of the region of interest, and change in sizeof the region of interest.
 5. The endoscope image processing deviceaccording to claim 3, wherein the processor performs the judgementprocessing based on a detection result obtained by detecting at leastany one of presence or absence of water feeding from the endoscope tothe region of interest, presence or absence of air feeding from theendoscope to the region of interest, and a moving direction of aninsertion portion to be inserted into the tubular portion in theendoscope.
 6. The endoscope image processing device according to claim4, wherein when the processor detects such change in position of theregion of interest that the region of interest is approaching anobstacle that covers a whole of the region of interest in theobservation image, the processor acquires a judgment result indicatingthat degradation of visibility of the region of interest included in theobservation image is predicted.
 7. The endoscope image processing deviceaccording to claim 4, wherein when the processor detects such change inposition of the region of interest that the region of interestapproaches an end of the observation image, the processor acquires ajudgement result indicating that degradation of visibility of the regionof interest included in the observation image is predicted.
 8. Theendoscope image processing device according to claim 5, wherein when theprocessor detects that the insertion portion moves in a direction inwhich the insertion portion is removed from an inside of the tubularportion, the processor acquires a judgement result indicating thatdegradation of visibility of the region of interest included in theobservation image is predicted.
 9. The endoscope image processing deviceaccording to claim 1, wherein the region of interest exists on an innerperipheral surface of the tubular portion.
 10. An endoscope imageprocessing method comprising: performing processing for detecting aregion of interest with respect to an observation image obtained byperforming image pickup of an object inside a tubular portion of anobject; performing judgment processing as to whether degradation ofvisibility of the region of interest included in the observation imageis predicted; and performing emphasis processing for emphasizing aposition of the region of interest when a judgement result indicatingthat the degradation of visibility of the region of interest included inthe observation image is predicted is acquired by the judgementprocessing.